{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.7 softmax回归的简洁实现\n",
    "\n",
    "我们在3.3节（线性回归的简洁实现）中已经了解了使用Tensorflow2.0实现模型的便利。下面，让我们再次使用Tensorflow2.0来实现一个softmax回归模型。首先导入所需的包或模块。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T11:07:44.653406Z",
     "start_time": "2020-03-29T11:07:41.554561Z"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 3.7.1 获取和读取数据\n",
    "\n",
    "我们仍然使用Fashion-MNIST数据集和上一节中设置的批量大小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对数据进行处理，归一化，便于训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train / 255.0\n",
    "x_test = x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 28, 28), (60000,))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在3.4节（softmax回归）中提到，softmax回归的输出层是一个全连接层。因此，我们添加一个输出个数为10的全连接层。 第一层是Flatten，将28 * 28的像素值，压缩成一行 (784, ) 第二层还是Dense，因为是多分类问题，激活函数使用softmax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.Sequential([keras.layers.Flatten(input_shape=(28, 28)), \n",
    "                         keras.layers.Dense(10, activation=tf.nn.softmax)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class Flatten in module tensorflow.python.keras.layers.core:\n",
      "\n",
      "class Flatten(tensorflow.python.keras.engine.base_layer.Layer)\n",
      " |  Flatten(data_format=None, **kwargs)\n",
      " |  \n",
      " |  Flattens the input. Does not affect the batch size.\n",
      " |  \n",
      " |  If inputs are shaped `(batch,)` without a channel dimension, then flattening\n",
      " |  adds an extra channel dimension and output shapes are `(batch, 1)`.\n",
      " |  \n",
      " |  Arguments:\n",
      " |    data_format: A string,\n",
      " |      one of `channels_last` (default) or `channels_first`.\n",
      " |      The ordering of the dimensions in the inputs.\n",
      " |      `channels_last` corresponds to inputs with shape\n",
      " |      `(batch, ..., channels)` while `channels_first` corresponds to\n",
      " |      inputs with shape `(batch, channels, ...)`.\n",
      " |      It defaults to the `image_data_format` value found in your\n",
      " |      Keras config file at `~/.keras/keras.json`.\n",
      " |      If you never set it, then it will be \"channels_last\".\n",
      " |  \n",
      " |  Example:\n",
      " |  \n",
      " |  ```python\n",
      " |  model = Sequential()\n",
      " |  model.add(Convolution2D(64, 3, 3,\n",
      " |                          border_mode='same',\n",
      " |                          input_shape=(3, 32, 32)))\n",
      " |  # now: model.output_shape == (None, 64, 32, 32)\n",
      " |  \n",
      " |  model.add(Flatten())\n",
      " |  # now: model.output_shape == (None, 65536)\n",
      " |  ```\n",
      " |  \n",
      " |  Method resolution order:\n",
      " |      Flatten\n",
      " |      tensorflow.python.keras.engine.base_layer.Layer\n",
      " |      tensorflow.python.module.module.Module\n",
      " |      tensorflow.python.training.tracking.tracking.AutoTrackable\n",
      " |      tensorflow.python.training.tracking.base.Trackable\n",
      " |      builtins.object\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __init__(self, data_format=None, **kwargs)\n",
      " |  \n",
      " |  call(self, inputs)\n",
      " |      This is where the layer's logic lives.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          inputs: Input tensor, or list/tuple of input tensors.\n",
      " |          **kwargs: Additional keyword arguments.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor or list/tuple of tensors.\n",
      " |  \n",
      " |  compute_output_shape(self, input_shape)\n",
      " |      Computes the output shape of the layer.\n",
      " |      \n",
      " |      If the layer has not been built, this method will call `build` on the\n",
      " |      layer. This assumes that the layer will later be used with inputs that\n",
      " |      match the input shape provided here.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          input_shape: Shape tuple (tuple of integers)\n",
      " |              or list of shape tuples (one per output tensor of the layer).\n",
      " |              Shape tuples can include None for free dimensions,\n",
      " |              instead of an integer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          An input shape tuple.\n",
      " |  \n",
      " |  get_config(self)\n",
      " |      Returns the config of the layer.\n",
      " |      \n",
      " |      A layer config is a Python dictionary (serializable)\n",
      " |      containing the configuration of a layer.\n",
      " |      The same layer can be reinstantiated later\n",
      " |      (without its trained weights) from this configuration.\n",
      " |      \n",
      " |      The config of a layer does not include connectivity\n",
      " |      information, nor the layer class name. These are handled\n",
      " |      by `Network` (one layer of abstraction above).\n",
      " |      \n",
      " |      Returns:\n",
      " |          Python dictionary.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Methods inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  __call__(self, inputs, *args, **kwargs)\n",
      " |      Wraps `call`, applying pre- and post-processing steps.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: input tensor(s).\n",
      " |        *args: additional positional arguments to be passed to `self.call`.\n",
      " |        **kwargs: additional keyword arguments to be passed to `self.call`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor(s).\n",
      " |      \n",
      " |      Note:\n",
      " |        - The following optional keyword arguments are reserved for specific uses:\n",
      " |          * `training`: Boolean scalar tensor of Python boolean indicating\n",
      " |            whether the `call` is meant for training or inference.\n",
      " |          * `mask`: Boolean input mask.\n",
      " |        - If the layer's `call` method takes a `mask` argument (as some Keras\n",
      " |          layers do), its default value will be set to the mask generated\n",
      " |          for `inputs` by the previous layer (if `input` did come from\n",
      " |          a layer that generated a corresponding mask, i.e. if it came from\n",
      " |          a Keras layer with masking support.\n",
      " |      \n",
      " |      Raises:\n",
      " |        ValueError: if the layer's `call` method returns None (an invalid value).\n",
      " |  \n",
      " |  __delattr__(self, name)\n",
      " |      Implement delattr(self, name).\n",
      " |  \n",
      " |  __getstate__(self)\n",
      " |  \n",
      " |  __setattr__(self, name, value)\n",
      " |      Support self.foo = trackable syntax.\n",
      " |  \n",
      " |  __setstate__(self, state)\n",
      " |  \n",
      " |  add_loss(self, losses, inputs=None)\n",
      " |      Add loss tensor(s), potentially dependent on layer inputs.\n",
      " |      \n",
      " |      Some losses (for instance, activity regularization losses) may be dependent\n",
      " |      on the inputs passed when calling a layer. Hence, when reusing the same\n",
      " |      layer on different inputs `a` and `b`, some entries in `layer.losses` may\n",
      " |      be dependent on `a` and some on `b`. This method automatically keeps track\n",
      " |      of dependencies.\n",
      " |      \n",
      " |      This method can be used inside a subclassed layer or model's `call`\n",
      " |      function, in which case `losses` should be a Tensor or list of Tensors.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      class MyLayer(tf.keras.layers.Layer):\n",
      " |        def call(inputs, self):\n",
      " |          self.add_loss(tf.abs(tf.reduce_mean(inputs)), inputs=True)\n",
      " |          return inputs\n",
      " |      ```\n",
      " |      \n",
      " |      This method can also be called directly on a Functional Model during\n",
      " |      construction. In this case, any loss Tensors passed to this Model must\n",
      " |      be symbolic and be able to be traced back to the model's `Input`s. These\n",
      " |      losses become part of the model's topology and are tracked in `get_config`.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      inputs = tf.keras.Input(shape=(10,))\n",
      " |      x = tf.keras.layers.Dense(10)(inputs)\n",
      " |      outputs = tf.keras.layers.Dense(1)(x)\n",
      " |      model = tf.keras.Model(inputs, outputs)\n",
      " |      # Actvity regularization.\n",
      " |      model.add_loss(tf.abs(tf.reduce_mean(x)))\n",
      " |      ```\n",
      " |      \n",
      " |      If this is not the case for your loss (if, for example, your loss references\n",
      " |      a `Variable` of one of the model's layers), you can wrap your loss in a\n",
      " |      zero-argument lambda. These losses are not tracked as part of the model's\n",
      " |      topology since they can't be serialized.\n",
      " |      \n",
      " |      Example:\n",
      " |      \n",
      " |      ```python\n",
      " |      inputs = tf.keras.Input(shape=(10,))\n",
      " |      x = tf.keras.layers.Dense(10)(inputs)\n",
      " |      outputs = tf.keras.layers.Dense(1)(x)\n",
      " |      model = tf.keras.Model(inputs, outputs)\n",
      " |      # Weight regularization.\n",
      " |      model.add_loss(lambda: tf.reduce_mean(x.kernel))\n",
      " |      ```\n",
      " |      \n",
      " |      The `get_losses_for` method allows to retrieve the losses relevant to a\n",
      " |      specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        losses: Loss tensor, or list/tuple of tensors. Rather than tensors, losses\n",
      " |          may also be zero-argument callables which create a loss tensor.\n",
      " |        inputs: Ignored when executing eagerly. If anything other than None is\n",
      " |          passed, it signals the losses are conditional on some of the layer's\n",
      " |          inputs, and thus they should only be run where these inputs are\n",
      " |          available. This is the case for activity regularization losses, for\n",
      " |          instance. If `None` is passed, the losses are assumed\n",
      " |          to be unconditional, and will apply across all dataflows of the layer\n",
      " |          (e.g. weight regularization losses).\n",
      " |  \n",
      " |  add_metric(self, value, aggregation=None, name=None)\n",
      " |      Adds metric tensor to the layer.\n",
      " |      \n",
      " |      Args:\n",
      " |        value: Metric tensor.\n",
      " |        aggregation: Sample-wise metric reduction function. If `aggregation=None`,\n",
      " |          it indicates that the metric tensor provided has been aggregated\n",
      " |          already. eg, `bin_acc = BinaryAccuracy(name='acc')` followed by\n",
      " |          `model.add_metric(bin_acc(y_true, y_pred))`. If aggregation='mean', the\n",
      " |          given metric tensor will be sample-wise reduced using `mean` function.\n",
      " |          eg, `model.add_metric(tf.reduce_sum(outputs), name='output_mean',\n",
      " |          aggregation='mean')`.\n",
      " |        name: String metric name.\n",
      " |      \n",
      " |      Raises:\n",
      " |        ValueError: If `aggregation` is anything other than None or `mean`.\n",
      " |  \n",
      " |  add_update(self, updates, inputs=None)\n",
      " |      Add update op(s), potentially dependent on layer inputs. (deprecated arguments)\n",
      " |      \n",
      " |      Warning: SOME ARGUMENTS ARE DEPRECATED: `(inputs)`. They will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      `inputs` is now automatically inferred\n",
      " |      \n",
      " |      Weight updates (for instance, the updates of the moving mean and variance\n",
      " |      in a BatchNormalization layer) may be dependent on the inputs passed\n",
      " |      when calling a layer. Hence, when reusing the same layer on\n",
      " |      different inputs `a` and `b`, some entries in `layer.updates` may be\n",
      " |      dependent on `a` and some on `b`. This method automatically keeps track\n",
      " |      of dependencies.\n",
      " |      \n",
      " |      The `get_updates_for` method allows to retrieve the updates relevant to a\n",
      " |      specific set of inputs.\n",
      " |      \n",
      " |      This call is ignored when eager execution is enabled (in that case, variable\n",
      " |      updates are run on the fly and thus do not need to be tracked for later\n",
      " |      execution).\n",
      " |      \n",
      " |      Arguments:\n",
      " |        updates: Update op, or list/tuple of update ops, or zero-arg callable\n",
      " |          that returns an update op. A zero-arg callable should be passed in\n",
      " |          order to disable running the updates by setting `trainable=False`\n",
      " |          on this Layer, when executing in Eager mode.\n",
      " |        inputs: Deprecated, will be automatically inferred.\n",
      " |  \n",
      " |  add_variable(self, *args, **kwargs)\n",
      " |      Deprecated, do NOT use! Alias for `add_weight`. (deprecated)\n",
      " |      \n",
      " |      Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      Please use `layer.add_weight` method instead.\n",
      " |  \n",
      " |  add_weight(self, name=None, shape=None, dtype=None, initializer=None, regularizer=None, trainable=None, constraint=None, partitioner=None, use_resource=None, synchronization=<VariableSynchronization.AUTO: 0>, aggregation=<VariableAggregation.NONE: 0>, **kwargs)\n",
      " |      Adds a new variable to the layer.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        name: Variable name.\n",
      " |        shape: Variable shape. Defaults to scalar if unspecified.\n",
      " |        dtype: The type of the variable. Defaults to `self.dtype` or `float32`.\n",
      " |        initializer: Initializer instance (callable).\n",
      " |        regularizer: Regularizer instance (callable).\n",
      " |        trainable: Boolean, whether the variable should be part of the layer's\n",
      " |          \"trainable_variables\" (e.g. variables, biases)\n",
      " |          or \"non_trainable_variables\" (e.g. BatchNorm mean and variance).\n",
      " |          Note that `trainable` cannot be `True` if `synchronization`\n",
      " |          is set to `ON_READ`.\n",
      " |        constraint: Constraint instance (callable).\n",
      " |        partitioner: Partitioner to be passed to the `Trackable` API.\n",
      " |        use_resource: Whether to use `ResourceVariable`.\n",
      " |        synchronization: Indicates when a distributed a variable will be\n",
      " |          aggregated. Accepted values are constants defined in the class\n",
      " |          `tf.VariableSynchronization`. By default the synchronization is set to\n",
      " |          `AUTO` and the current `DistributionStrategy` chooses\n",
      " |          when to synchronize. If `synchronization` is set to `ON_READ`,\n",
      " |          `trainable` must not be set to `True`.\n",
      " |        aggregation: Indicates how a distributed variable will be aggregated.\n",
      " |          Accepted values are constants defined in the class\n",
      " |          `tf.VariableAggregation`.\n",
      " |        **kwargs: Additional keyword arguments. Accepted values are `getter`,\n",
      " |          `collections`, `experimental_autocast` and `caching_device`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        The created variable. Usually either a `Variable` or `ResourceVariable`\n",
      " |        instance. If `partitioner` is not `None`, a `PartitionedVariable`\n",
      " |        instance is returned.\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called with partitioned variable regularization and\n",
      " |          eager execution is enabled.\n",
      " |        ValueError: When giving unsupported dtype and no initializer or when\n",
      " |          trainable has been set to True with synchronization set as `ON_READ`.\n",
      " |  \n",
      " |  apply(self, inputs, *args, **kwargs)\n",
      " |      Deprecated, do NOT use! (deprecated)\n",
      " |      \n",
      " |      Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version.\n",
      " |      Instructions for updating:\n",
      " |      Please use `layer.__call__` method instead.\n",
      " |      \n",
      " |      This is an alias of `self.__call__`.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor(s).\n",
      " |        *args: additional positional arguments to be passed to `self.call`.\n",
      " |        **kwargs: additional keyword arguments to be passed to `self.call`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor(s).\n",
      " |  \n",
      " |  build(self, input_shape)\n",
      " |      Creates the variables of the layer (optional, for subclass implementers).\n",
      " |      \n",
      " |      This is a method that implementers of subclasses of `Layer` or `Model`\n",
      " |      can override if they need a state-creation step in-between\n",
      " |      layer instantiation and layer call.\n",
      " |      \n",
      " |      This is typically used to create the weights of `Layer` subclasses.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        input_shape: Instance of `TensorShape`, or list of instances of\n",
      " |          `TensorShape` if the layer expects a list of inputs\n",
      " |          (one instance per input).\n",
      " |  \n",
      " |  compute_mask(self, inputs, mask=None)\n",
      " |      Computes an output mask tensor.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          inputs: Tensor or list of tensors.\n",
      " |          mask: Tensor or list of tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |          None or a tensor (or list of tensors,\n",
      " |              one per output tensor of the layer).\n",
      " |  \n",
      " |  compute_output_signature(self, input_signature)\n",
      " |      Compute the output tensor signature of the layer based on the inputs.\n",
      " |      \n",
      " |      Unlike a TensorShape object, a TensorSpec object contains both shape\n",
      " |      and dtype information for a tensor. This method allows layers to provide\n",
      " |      output dtype information if it is different from the input dtype.\n",
      " |      For any layer that doesn't implement this function,\n",
      " |      the framework will fall back to use `compute_output_shape`, and will\n",
      " |      assume that the output dtype matches the input dtype.\n",
      " |      \n",
      " |      Args:\n",
      " |        input_signature: Single TensorSpec or nested structure of TensorSpec\n",
      " |          objects, describing a candidate input for the layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Single TensorSpec or nested structure of TensorSpec objects, describing\n",
      " |          how the layer would transform the provided input.\n",
      " |      \n",
      " |      Raises:\n",
      " |        TypeError: If input_signature contains a non-TensorSpec object.\n",
      " |  \n",
      " |  count_params(self)\n",
      " |      Count the total number of scalars composing the weights.\n",
      " |      \n",
      " |      Returns:\n",
      " |          An integer count.\n",
      " |      \n",
      " |      Raises:\n",
      " |          ValueError: if the layer isn't yet built\n",
      " |            (in which case its weights aren't yet defined).\n",
      " |  \n",
      " |  get_input_at(self, node_index)\n",
      " |      Retrieves the input tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor (or list of tensors if the layer has multiple inputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_input_mask_at(self, node_index)\n",
      " |      Retrieves the input mask tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A mask tensor\n",
      " |          (or list of tensors if the layer has multiple inputs).\n",
      " |  \n",
      " |  get_input_shape_at(self, node_index)\n",
      " |      Retrieves the input shape(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A shape tuple\n",
      " |          (or list of shape tuples if the layer has multiple inputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_losses_for(self, inputs)\n",
      " |      Retrieves losses relevant to a specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor or list/tuple of input tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |        List of loss tensors of the layer that depend on `inputs`.\n",
      " |  \n",
      " |  get_output_at(self, node_index)\n",
      " |      Retrieves the output tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A tensor (or list of tensors if the layer has multiple outputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_output_mask_at(self, node_index)\n",
      " |      Retrieves the output mask tensor(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A mask tensor\n",
      " |          (or list of tensors if the layer has multiple outputs).\n",
      " |  \n",
      " |  get_output_shape_at(self, node_index)\n",
      " |      Retrieves the output shape(s) of a layer at a given node.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          node_index: Integer, index of the node\n",
      " |              from which to retrieve the attribute.\n",
      " |              E.g. `node_index=0` will correspond to the\n",
      " |              first time the layer was called.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A shape tuple\n",
      " |          (or list of shape tuples if the layer has multiple outputs).\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |  \n",
      " |  get_updates_for(self, inputs)\n",
      " |      Retrieves updates relevant to a specific set of inputs.\n",
      " |      \n",
      " |      Arguments:\n",
      " |        inputs: Input tensor or list/tuple of input tensors.\n",
      " |      \n",
      " |      Returns:\n",
      " |        List of update ops of the layer that depend on `inputs`.\n",
      " |  \n",
      " |  get_weights(self)\n",
      " |      Returns the current weights of the layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Weights values as a list of numpy arrays.\n",
      " |  \n",
      " |  set_weights(self, weights)\n",
      " |      Sets the weights of the layer, from Numpy arrays.\n",
      " |      \n",
      " |      Arguments:\n",
      " |          weights: a list of Numpy arrays. The number\n",
      " |              of arrays and their shape must match\n",
      " |              number of the dimensions of the weights\n",
      " |              of the layer (i.e. it should match the\n",
      " |              output of `get_weights`).\n",
      " |      \n",
      " |      Raises:\n",
      " |          ValueError: If the provided weights list does not match the\n",
      " |              layer's specifications.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  from_config(config) from builtins.type\n",
      " |      Creates a layer from its config.\n",
      " |      \n",
      " |      This method is the reverse of `get_config`,\n",
      " |      capable of instantiating the same layer from the config\n",
      " |      dictionary. It does not handle layer connectivity\n",
      " |      (handled by Network), nor weights (handled by `set_weights`).\n",
      " |      \n",
      " |      Arguments:\n",
      " |          config: A Python dictionary, typically the\n",
      " |              output of get_config.\n",
      " |      \n",
      " |      Returns:\n",
      " |          A layer instance.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.keras.engine.base_layer.Layer:\n",
      " |  \n",
      " |  activity_regularizer\n",
      " |      Optional regularizer function for the output of this layer.\n",
      " |  \n",
      " |  dtype\n",
      " |  \n",
      " |  dynamic\n",
      " |  \n",
      " |  inbound_nodes\n",
      " |      Deprecated, do NOT use! Only for compatibility with external Keras.\n",
      " |  \n",
      " |  input\n",
      " |      Retrieves the input tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one input,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input tensor or list of input tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |        RuntimeError: If called in Eager mode.\n",
      " |        AttributeError: If no inbound nodes are found.\n",
      " |  \n",
      " |  input_mask\n",
      " |      Retrieves the input mask tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one inbound node,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input mask tensor (potentially None) or list of input\n",
      " |          mask tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer is connected to\n",
      " |          more than one incoming layers.\n",
      " |  \n",
      " |  input_shape\n",
      " |      Retrieves the input shape(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one input,\n",
      " |      i.e. if it is connected to one incoming layer, or if all inputs\n",
      " |      have the same shape.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Input shape, as an integer shape tuple\n",
      " |          (or list of shape tuples, one tuple per input tensor).\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer has no defined input_shape.\n",
      " |          RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  input_spec\n",
      " |  \n",
      " |  losses\n",
      " |      Losses which are associated with this `Layer`.\n",
      " |      \n",
      " |      Variable regularization tensors are created when this property is accessed,\n",
      " |      so it is eager safe: accessing `losses` under a `tf.GradientTape` will\n",
      " |      propagate gradients back to the corresponding variables.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of tensors.\n",
      " |  \n",
      " |  metrics\n",
      " |  \n",
      " |  name\n",
      " |      Returns the name of this module as passed or determined in the ctor.\n",
      " |      \n",
      " |      NOTE: This is not the same as the `self.name_scope.name` which includes\n",
      " |      parent module names.\n",
      " |  \n",
      " |  non_trainable_variables\n",
      " |  \n",
      " |  non_trainable_weights\n",
      " |  \n",
      " |  outbound_nodes\n",
      " |      Deprecated, do NOT use! Only for compatibility with external Keras.\n",
      " |  \n",
      " |  output\n",
      " |      Retrieves the output tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one output,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |        Output tensor or list of output tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |        AttributeError: if the layer is connected to more than one incoming\n",
      " |          layers.\n",
      " |        RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  output_mask\n",
      " |      Retrieves the output mask tensor(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has exactly one inbound node,\n",
      " |      i.e. if it is connected to one incoming layer.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Output mask tensor (potentially None) or list of output\n",
      " |          mask tensors.\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer is connected to\n",
      " |          more than one incoming layers.\n",
      " |  \n",
      " |  output_shape\n",
      " |      Retrieves the output shape(s) of a layer.\n",
      " |      \n",
      " |      Only applicable if the layer has one output,\n",
      " |      or if all outputs have the same shape.\n",
      " |      \n",
      " |      Returns:\n",
      " |          Output shape, as an integer shape tuple\n",
      " |          (or list of shape tuples, one tuple per output tensor).\n",
      " |      \n",
      " |      Raises:\n",
      " |          AttributeError: if the layer has no defined output shape.\n",
      " |          RuntimeError: if called in Eager mode.\n",
      " |  \n",
      " |  stateful\n",
      " |  \n",
      " |  trainable\n",
      " |  \n",
      " |  trainable_variables\n",
      " |      Sequence of trainable variables owned by this module and its submodules.\n",
      " |      \n",
      " |      Note: this method uses reflection to find variables on the current instance\n",
      " |      and submodules. For performance reasons you may wish to cache the result\n",
      " |      of calling this method if you don't expect the return value to change.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A sequence of variables for the current module (sorted by attribute\n",
      " |        name) followed by variables from all submodules recursively (breadth\n",
      " |        first).\n",
      " |  \n",
      " |  trainable_weights\n",
      " |  \n",
      " |  updates\n",
      " |  \n",
      " |  variables\n",
      " |      Returns the list of all layer variables/weights.\n",
      " |      \n",
      " |      Alias of `self.weights`.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of variables.\n",
      " |  \n",
      " |  weights\n",
      " |      Returns the list of all layer variables/weights.\n",
      " |      \n",
      " |      Returns:\n",
      " |        A list of variables.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Class methods inherited from tensorflow.python.module.module.Module:\n",
      " |  \n",
      " |  with_name_scope(method) from builtins.type\n",
      " |      Decorator to automatically enter the module name scope.\n",
      " |      \n",
      " |      ```\n",
      " |      class MyModule(tf.Module):\n",
      " |        @tf.Module.with_name_scope\n",
      " |        def __call__(self, x):\n",
      " |          if not hasattr(self, 'w'):\n",
      " |            self.w = tf.Variable(tf.random.normal([x.shape[1], 64]))\n",
      " |          return tf.matmul(x, self.w)\n",
      " |      ```\n",
      " |      \n",
      " |      Using the above module would produce `tf.Variable`s and `tf.Tensor`s whose\n",
      " |      names included the module name:\n",
      " |      \n",
      " |      ```\n",
      " |      mod = MyModule()\n",
      " |      mod(tf.ones([8, 32]))\n",
      " |      # ==> <tf.Tensor: ...>\n",
      " |      mod.w\n",
      " |      # ==> <tf.Variable ...'my_module/w:0'>\n",
      " |      ```\n",
      " |      \n",
      " |      Args:\n",
      " |        method: The method to wrap.\n",
      " |      \n",
      " |      Returns:\n",
      " |        The original method wrapped such that it enters the module's name scope.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.module.module.Module:\n",
      " |  \n",
      " |  name_scope\n",
      " |      Returns a `tf.name_scope` instance for this class.\n",
      " |  \n",
      " |  submodules\n",
      " |      Sequence of all sub-modules.\n",
      " |      \n",
      " |      Submodules are modules which are properties of this module, or found as\n",
      " |      properties of modules which are properties of this module (and so on).\n",
      " |      \n",
      " |      ```\n",
      " |      a = tf.Module()\n",
      " |      b = tf.Module()\n",
      " |      c = tf.Module()\n",
      " |      a.b = b\n",
      " |      b.c = c\n",
      " |      assert list(a.submodules) == [b, c]\n",
      " |      assert list(b.submodules) == [c]\n",
      " |      assert list(c.submodules) == []\n",
      " |      ```\n",
      " |      \n",
      " |      Returns:\n",
      " |        A sequence of all submodules.\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors inherited from tensorflow.python.training.tracking.base.Trackable:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables (if defined)\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object (if defined)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(keras.layers.Flatten)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7.3 softmax和交叉熵损失函数\n",
    "\n",
    "如果做了上一节的练习，那么你可能意识到了分开定义softmax运算和交叉熵损失函数可能会造成数值不稳定。因此，Tensorflow2.0的keras API提供了一个loss参数。它的数值稳定性更好。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = 'sparse_categorical_crossentropy'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7.4 定义优化算法\n",
    "\n",
    "我们使用学习率为0.1的小批量随机梯度下降作为优化算法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.SGD(0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7.5 训练模型\n",
    "\n",
    "接下来，我们使用上一节中定义的训练函数来训练模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 3s 47us/sample - loss: 0.4798 - accuracy: 0.8371\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 3s 44us/sample - loss: 0.4604 - accuracy: 0.8430\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 3s 42us/sample - loss: 0.4495 - accuracy: 0.8462\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 3s 44us/sample - loss: 0.4431 - accuracy: 0.8475\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 3s 48us/sample - loss: 0.4358 - accuracy: 0.8504\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f4bef0616d0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=5, batch_size=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，比较模型在测试数据集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 87us/sample - loss: 0.4638 - accuracy: 0.8386\n",
      "Test Acc: 0.8386\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(x_test, y_test)\n",
    "print('Test Acc:',test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方式二"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SparseCategoricalCrossentropy 已经除以了 batch_size,所以sgd 不需要再除以 batch_size 了\n",
    "# https://stackoverflow.com/questions/58159154/how-to-calculate-categorical-cross-entropy-by-hand\n",
    "# 默认，SUM_OVER_BATCH_SIZE，Scalar SUM divided by number of elements in losses. \n",
    "cross_entropy = losses.SparseCategoricalCrossentropy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T11:08:06.650505Z",
     "start_time": "2020-03-29T11:08:06.525862Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float64, numpy=-0.4570695745672833>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "# 2 个样本3个类别的预测概率\n",
    "y_hat = np.array([[.9, .05, .05], [.5, .3, .2], [.05, .01, .94]])\n",
    "# 2 个样本真实值\n",
    "y = np.array([0, 1, 2], dtype='int32')\n",
    "tf.reduce_sum(tf.math.log(tf.boolean_mask(y_hat, tf.one_hot(y, depth=3))))/3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-29T11:08:09.292644Z",
     "start_time": "2020-03-29T11:08:09.285525Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss:  0.45706955\n"
     ]
    }
   ],
   "source": [
    "cce = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "loss = cce(\n",
    "  tf.convert_to_tensor([0, 1, 2]),\n",
    "  tf.convert_to_tensor([[.9, .05, .05], [.5, .3, .2], [.05, .01, .94]]))\n",
    "print('Loss: ', loss.numpy())  # Loss: 0.3239"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 迭代周期，学习率\n",
    "num_epochs, lr = 5, 0.1\n",
    "def train_ch3(net, train_iter, test_iter, loss, num_epochs, batch_size, params=None, lr=None, trainer=None):\n",
    "    for epoch in range(num_epochs):\n",
    "        train_l_sum, train_acc_sum, n = 0.0, 0.0, 0\n",
    "        for X, y in train_iter:\n",
    "            with tf.GradientTape() as tape:\n",
    "                y_hat = net(X, training=True)\n",
    "                l = loss(y, y_hat)\n",
    "                #print(params)\n",
    "                grads = tape.gradient(l, net.trainable_variables)\n",
    "                # tf.keras.optimizers.SGD 直接使用是随机梯度下降 theta(t+1) = theta(t) - learning_rate * gradient\n",
    "                trainer.apply_gradients(zip(grads, net.trainable_variables))  \n",
    "                \n",
    "            train_l_sum += l.numpy()\n",
    "            train_acc_sum += tf.reduce_sum(tf.cast(tf.argmax(y_hat, axis=1) == tf.cast(y, dtype=tf.int64), dtype=tf.int64)).numpy()\n",
    "            n += y.shape[0]\n",
    "        test_acc = evaluate_accuracy(test_iter, net)\n",
    "        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f'% (epoch + 1, train_l_sum / n, train_acc_sum / n, test_acc))\n",
    "\n",
    "trainer = tf.keras.optimizers.SGD(lr)\n",
    "train_ch3(model, train_iter, test_iter, cross_entropy, num_epochs, batch_size, lr=lr, trainer=trainer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 小结\n",
    "\n",
    "- Tensorflow2.0提供的函数往往具有更好的数值稳定性。\n",
    "- 可以使用Tensorflow2.0更简洁地实现softmax回归。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
